Skip to content

Add COMPUTER_USE_ABLATE mechanism ablation switches - #5

Closed
Haoqing Wang (hqhq1025) wants to merge 13 commits into
fix/linux-ax-input-and-errors-20261116from
feature/hcu-ablation-switches-20261116
Closed

Haoqing Wang (hqhq1025) wants to merge 13 commits into
fix/linux-ax-input-and-errors-20261116from
feature/hcu-ablation-switches-20261116

Conversation

@hqhq1025

Copy link
Copy Markdown
Collaborator

This pull request is stacked on #4 and targets that branch; I will retarget it to main after the fixes merge.

It adds COMPUTER_USE_ABLATE, a comma-separated switch that turns off one or more HCU mechanisms so that their effect can be measured. The default (unset) behavior is unchanged.

  • batching: an act call accepts one input step.
  • wait: wait steps are rejected and the model is sent to a fresh read.
  • diff: the change report is replaced by the whole tree.
  • guard: the input guards run in shadow mode. They log what they would block and let the input through. This needs COMPUTER_USE_GUARD_LOG.

An ablated server also stops describing what it removed. The server instructions, tool definitions, error results and the generated guides are checked for the words of each removed mechanism, and the server refuses to start on a misspelled switch.

The OSWorld runner takes --ablate, records the setting in its results, probes that the guest server applies it, and exports the guard log. configs/ablation holds the six OSWorld 1.0 groups.

Tests

  • make test as in the fix branch (the same six test_setup_canary environment errors).
  • scripts/maint/check_ablation_text.py passes for every subset of the four switches and for the invalid values.

Parse and validate batching, wait, diff and guard; refuse to start on an
unknown value, on guard outside Linux, and on guard without
COMPUTER_USE_GUARD_LOG. The schema, tool descriptions and server
instructions drop the fields and sentences of ablated mechanisms; with no
ablation they are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
batching: an AX act takes one input step; wait steps may surround it
(rule one_input_step). wait: wait steps are no longer an action type, and
error texts no longer suggest them. diff: act runs as before but returns
only its execution evidence; its reads are not recorded as delivered, so
a later observe compares with what the model actually saw.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With guard ablated, the AX target identity, batch window, point owner and
geometry checks and the GUI layout and zoom checks still decide, but only
log the input they would stop to COMPUTER_USE_GUARD_LOG and let it
through. A path with no live node still fails as stale.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--ablate (config key ablate) selects the removed mechanisms, passes them
to the MCP server env of all four harnesses, gives the agent guides
without the passages of the removed mechanisms, and records the set in
result.json and meta.json. With guard ablated the runner collects the
guest's guard log. configs/ablation holds the six OSWorld 1.0 groups.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The semantic refusal fallback joins the other resolution code in
element_resolver.py, and the GUI layout log line is built in
guard_shadow.py, so the dispatch modules stay near the size limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Go: parsing and startup refusal, model-visible text of every ablation
set, valid description examples, schema and decoder agreement under each
ablation, the one-input-step rule, wait removal, act results without
feedback, and the guard log. Provider: shadow mode logs and lets input
through for target identity, batch window, point owner and GUI layout,
and changes nothing without the ablation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
check_ablation_text.py starts the server for each COMPUTER_USE_ABLATE
subset, reads instructions, tools and rejected calls, generates the
guides, and fails on words or act examples of a removed mechanism.
check_all.sh runs it with the candidate binary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Deleting the passages of a removed mechanism left the agent to learn the
changed rules from rejected calls. The guide now adds one sentence per
ablation (batching, wait, diff), matching the AX tool description, and no
new advice; guard adds none. Everything else in the guide is still only
removed text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…left

guard: the focus check of a focus-bound batch, the select range end and
the combobox edit partner now go through the shadow, and a shadowed
target keeps the window that holds it. The full arm no longer reads the
tree for a shadow it does not use.

wait: a rejected wait step points to a fresh observe instead of an input
action guessed from its fields, and the no-tree window text no longer
offers a wait.

batching: input after close is refused as close-not-last without
suggesting a wait, and no_focused_window no longer says to start a batch
with activate.

diff: act errors no longer carry a new wN, the values and windows a wait
saw, or a promised observation; the tree reference no longer mentions
diffs.

A wait-only act without ref or window_ref no longer panics the server
when the window list cannot be read. This crash predates the switches and
hit every arm that keeps wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
batching removes "start the batch with activate"; diff removes the
wait_timeout advice to read the returned state. check_ablation_text.py
looks for both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The preflight probe sends one call per removed mechanism and requires
each to be refused, and requires the server to refuse an unknown
COMPUTER_USE_ABLATE value, so a build without the switches fails
preparation instead of running full HCU under an ablation label.
meta.json records the MCP binary's SHA-256. A task now fails before its
agent starts when the gateway writes another token log than the run
reads, which happened when ablation groups shared one gateway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removing diff now answers what the diff contributes: an act still
returns the state after its actions, as the whole current tree of its
window, in place of the effect and windows report, the diff and the
notes that compare with an earlier read. observe always repeats the
tree instead of "Tree unchanged". The tree is read with the same
post-action stabilization as a diff, so the ablation changes what is
shown, not when it is read.

The earlier variant returned only the execution evidence, which also
removed every post-act observation; the error-path trimming it needed
is reverted. effect_unknown stays in the evidence and a settle wait
keeps its message: the first is execution evidence, the second belongs
to wait.

The guide and tool description say that an act result shows the whole
tree instead of a report of what changed, and keep the advice to read
the returned state. Without an ablation every model-visible byte is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove the batch-window sentence when batching is ablated, assert that the
diff-ablation error keeps its cause, and check that read_text survives every
ablation set.
@hqhq1025

Copy link
Copy Markdown
Collaborator Author

Superseded by #7. The fixes in #4 were rebase-merged, so the ablation commits are rebased onto the new main; the final content is unchanged.

@hqhq1025
Haoqing Wang (hqhq1025) deleted the feature/hcu-ablation-switches-20261116 branch November 16, 2026 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant